MASTERS THESIS Functional Semantic Analysis of Web Pages on the Visual Layer

This masters thesis is motivated by the fact that data records on web pages are structured not only by word content but also by an implied visual hierarchy. A model of this visual hierarchy can greatly support automatic information extraction approaches become more domain independent and robust against variations of HTML syntax changes because the representation of information on the visual layer has to remain rather constant so as to remain understandable by humans. We refer to this visual layer as functional level which expresses the functional support for humans when structuring information visually. This masters thesis first gives a thorough literature overview on (visual) document analysis and then presents such a functional level record detection system named REDEVILA (REcord DEtection on the VIsual LAyer). The approach works by superimposing a multi-topological grid onto the visual layer of web pages serving as an efficient spatial reasoning data structure for detecting the functional semantics between data items or data records. The system is principally domain independent as long as the layout hierarchy provided by the web page mainly depends on general topological and geometrical characteristics such as font size, distance and indention and not on color properties or word semantics. We further propose a novel diagonal ordering scheme to obtain a more “natural” or human-intuitive ordering and demonstrate the concept and problems of the visual based detection of single records. For the experimental evaluation we selected web pages from four different domains (blogs, search results, personal homepages and online newspapers) to show the basic domain independence of our system. Experiments were performed on 85 web pages and achieved a fair overall performance. We conclude that, while in its early stages, the visual approach has the potential to significantly improve the performance and robustness of traditional wrapper systems to induce a higher level of generalization and represent a next step towards generic web wrapping.